Visual Analytics

3.2.1 Using Zathura

3.2.2 Question 1

1. Problem
   

   Visit the website: http://www.visualcomplexity.com/vc/. Have a close look at the available visualisation techniques. In your opinion, which techniques are among the most useful? Or which one is the most pretty visual display? Explain and justify your preference. Note: there is no right or wrong answer in this question.

2. Working
   

   This Visual from the Google AI blog highlighting the important features detected by a neural network is a unique insight into the predictive modelling technique. It is not easy to understand the behaviour of a Neural Network and this visual offers a uniqe insight that could not easily be understood:

   

   This Visual however is arguably one of the most visually striking because of the vibrant colour choice:

   

3.2.3 Question 2

1. Problem
   

   Explore the online demo: http://graphs.gapminder.org/world/. What have you discovered or found from the visualization of the “Wealth and Health of Nations data set”? E.g. is there any correlation between GDP and Life Expectancy? etc.

2. Working
   

   There is a clear positive correlation between income and life expectancy, other features that are readily observable are:

   1. Regions
      

      It can be observed that Europe is a wealthy region (Yellow) while Africa and Asia are poor regions (cyan, red))

      This could be further observed to be a function of distance from the equator, generally regions lose to the equator like Europe, UK, USA are quite wealthy where as regions nearer the equator are more likely to be impovrished.

      Exceptions to this are:

      • Australia due to the rich resources such as coal/uranium and the head start in education owing to the European Descent.
      • Russia is impovrished because the October Revolutions that followed the first

      World War greatly modified there economy relative to the rest of Europe, this decayed into authoritarianism and eventually extreme wealth inequality (with the rise of homelessness and oligarchs) following the collapse of the Soviet Union at the end of the cold-war.

      This could be a functoin of temperature, a higher temperature may lead to :

      • increased spread of bacterial disease
      • increased perishability of food
   2. Population
      

      It isn't possible to say much about the influence population has because the population is measured by region without taking account of size (e.g. China is quite large while Vietnam is relatively small) or habitability (e.g. Japan is very mountainous and hard to farm or ranch on).

      This is made more difficult because the income is measured per person but the population is not the relative space that person might enjoy (i.e. the population density), the GDP per country rather than capita might be more instructive.

   3. Time Trends
      

      It appears that Europe (and Japan) have always enjoyed a higher GDP per capita while asia and Africa appear to lag in this respect.

      Other than the "reshuffling" caused by the Great Depression, WW I, WW II and the collapse of the soviet union Europe has always enjoyed a higher GDP hence a higher life expectancy.

      During the first two world wars (with the exception of Russia and Poland respectively, for obvious reasons) it appears that the influence of currency on life expectancy became even stronger, somewhat unfortunate for other regions given that these were European wars. The great depression also showed a similar effect on this correlation, going from a linear trend to an exponential.

   4. China as an Exception
      

      China appears to not follow a correlation between GDP and Life Expectancy until the new milennia, life expectancy rises following the second world war despite no change in GDP per Capita.

      In the 70's China begins to follow this trend moving in a diagonal fashion indicative of a correlation, this is presumably due to manufacturing exported to China perhaps due to the development of the micro-processor in the US.

      In the 2000's China began to improve GDP per capita more significantly, perhaps due to free market policies, and followed a trend where GDP per capita would strongly correlated with life expectancy.

3.2.4 Question 3

1. Problem
   

   Using a search engine, explore 3 applications/projects/tools that use visualisations. You are required to write a short summary about the applications/projects/tools. Why do you think they are significant? What are good and bad about these applications/projects/tools? Etc.

2. Working
   
   1. Gnome's Baobab / WinDirStat / KDirStat
      

      Gnome' Baobab, KDirStat and WinDirStat are disk usage analysers for Gnome, KDE and Windows respectively.

      

      

      1. Significance
         

         All three tools use a descending list of directories with a bar chart indicating the proportion of disk space consumed by that directory, to the right is a graphic shosing this distribution.

      2. Good Design
         

         All three tools have correctly implement an ordered list and bar chart, making it easy to understand and manage disk space on a system.

         Gnome's Baobab uses a ring chart with a popup overlay to describe the corresponding directory.

         This choice of graph makes it easy to understand which directories are consuming the most space while still having an overview of the structure of the directories, the popup prevents the graph from becoming too busy and showing unnecessary information.

         The ring chart will also re-centre following a selection of a directory allowing deeply nested structures to be understood easily.

      3. Bad Design
         

         Unfourtunately KDirstat and Windirstat only offer treemap visualisations, this choice of graph is vastly inferior to a ring chart because it can only clearly show a certain amount of information at an overview, it is difficult do understand deeply nested folder structures they won't re-generate the plot without rescanning the drive.

   2. GitHub Visualizer
      

      http://ghv.artzub.com/#repo=ranger-assets&climit=100&user=ranger

      The Github Visualiser is a way of visualising the proportionate activity of various repositiories of various projects.

      1. Significance
         

         This provides a novel way to understand the:

         • relative popularity of repos
         • The language most composing a repo / project
         • How Active a project is generally and over time.
      2. Good Design
         

         Using Bubbles to illustrate the is the overall share of a repo in a project provides a quick understanding at a glance.

         Using a Time Series Chart over time is an easy way to show trends.

      3. Bad Design
         

         The visualisation is far too busy to understand what is going on, for instance the size of the bubbles are not made clear, are they popularity, size, frequency of commits or frequency of clones / pulls of the repo?

         For instance the visualisatoin of one of my favourite pieces of software Ranger is such that I can infer nothing from it other than the fact that the web page is written in HTML and the project is written in python, which is totally obvious and not particularly helpful to get a deeper understanding of the Ranger project as opposed to say the Midnight Commander Project, although the visualisation for Midnight Commander is significantly better and so this may be a scaling issue.

   3. Cancer Graphic
      

      http://web.archive.org/web/20140526084103/http://www.cancerresearchuk.org:80/cancer-info/cancerstats/causes/attributable-risk/visualisation/

      This Plot attached here shows the attributes most likely to cause cancer:

      cd /tmp
      #wget "http://web.archive.org/web/20140801035734/http://publications.cancerresearchuk.org/downloads/product/CS_POSTER_ATTRIB.pdf"
      command -v pdftoppm >/dev/null 2>&1 || { echo >&2 "command -v foo >/dev/null 2>&1 || { echo >&2 "I require pdftoppm but its not installed.  Aborting."; exit 1; }I require foo but it's not installed.  Aborting."; exit 1; }
      pdftoppm CS_POSTER_ATTRIB.pdf  CS_POSTER_ATTRIB -png
      mv CS_POSTER*png ~/Notes/Org/Attachments/Statistics/
      ls ~/Notes/Org/Attachments/Statistics/CS_POSTER_ATT*png

      

      1. Significance
         

         This visualisation is significant because it effectively describes both the magnitude and interaction of various risk factors and behaviours on the probability of developing cancer.

         Statistically this can be a very difficult thing to describe and explain but this graphic very clearly shows what to look out or

      2. Good Design
         

         Having a key in the centre of the graph makes it very easy to determine what the individual elements of the visualisation mean.

         The relative size of individual risk factors means that at a galnce it is very easy to determine risk factors for cancer.

      3. Bad Design
         

         The graph unfourtunately is a little busy but this doesn't appear to be in a way that is disproportionate to the amount of information conveyed.

      4. Inferences
         

         An interesting component of this visualisation is that it clearly shows interactions of various items, for example the following are large to moderate risk factors for cancer generaly:

         • Overweight (BMI > 25 kg / m\(^2\), distninct from obesity which is $≈ > 30)
         • Inactivity

         While HRT is a very small risk factor for all types of cancer.

         This clearly illustrates that HRT may be medically appropriate in men despite the historical belive that such treamtent was correlated with prostate cancer in men (Bell et al. 2018) . Recent studies suggest that this might not be the case (Loeb et al. 2017) and HRT is (somewhat obviously) correlated with increased physical activity, weight loss (Traish 2014) and general health metrics (Saad et al. 2020) .

         This visualisation clearly shows that this treatment might generally lower the risk of cancer in men and provides a simple way to convey complex interactions between attributes and the complex statistics involved in this type of research in a way that clearly shows the important facts of the matter..

         Moreover this plot also only shows HRT as a risk factor in cancers that tend to overwhelmingly affect women, a distinction that is very important in that area of medical science.

3.2.5 Question 4

1. Problem
   

   For the following visualisations, in your opinion, are they good or bad; Justify the answer:

2. Working
   
   1. Plot A
      

      

      This plot is overly busy but also not very descriptive, meaning at a glance it is not possible to determine what the differences between different regions actually are.

      Upon closer inspection it is not clear what the bar charts are trying to illustrate, clearly the venn diagram is trying to illustrate the proportion of communication and overlap, relative to thhe number of users. This is an interesting way to try and show the interactions between the different communication strategies but the choice of a Venn diagram was misguided. A venn diagram can only represent 3 sets with circles ¹ and this has seemingly artificially restricted the number of potential behavriou types that the plot has tried to convey to such an extent that they may as well be disregarded entirely because they are just not descriptive enough to understand.

      Interestingly Eastern countries use a combination of content/messges while western countries tend to use or the other, this could be related to the pictographical written laguage implemented by Eastern / Asian Countries and could lead to insights on human behaviour, unfourtunately the plot does not clearly describe the meaning of the colours and so no inferences can be easily made.

      Moreover the size of the circles are relative to the total number of users, this is somewhat misleading because the plot shows that 33% of the USA uses Social Media while only 15% of China uses social media, countries/regions are not uniformally distributed or constructed so choosing to use regions as a delimiter when using absolute size of users is potentially misleading. This is partially addressed however by the Global Social Network penetration bar chart below the plot.

   2. Plot B
      

      

      • The amount of text overlayed on this graph makes it difficult to read.
      • The Growth rate of the publication should not be represented as a line because

      it indicates some type of continuous connection between the universities, it should be represented as either another column or ideally a seperate time series plot should be produced:

      • Such a plot should show the market share and allow the growth rate to be

      interpreted from the slope of the line in an organic fashion, that way the different universities could have both market share and growth rate compared between each other without trying to mentally visualise a cumullative summation.

3.2.6 Question 5

1. Problem
   

   Optional: If you still have time, play around this website to see how visualization help to find patterns: http://rocs.hu-berlin.de/viz/sgb/ (Coronavirus Geographic and Network visualisations)

2. Working
   

   :HideRef:

4.2.1 All Exercises

4.2.2 Question 1

1. Perception
   

   Perception is identifying and understanding a sensory input, such as perceiving:

   • an angle as obtuse or acute
   • an object to be round or sharp by touch or sight
   • an accent to be British, German, Chinese etc. by sound.
   • an object to be falling off the table.

   Perception is inherently dependent on experience, for example a person without exposure may not be able to perceive differences in accent.

   It is unlikely that two different people will perceive things in the exact same way.

   Generally perception is not something done consciously.

2. Cognition
   

   Cognition is processing information in an analytical way, working through a problem using prior training, it is done consciously.

   Examples of cognition include:

   • Determing the shape of a plot follows a certain model (e.g. polynomial, exponential/logarithmic, ARIMA, White Noise, etc.)
   • Recognising an object to be a 12 mm nut from brake caliper as opposed to a small metallic object.
   • Deciding that a person probably spent time in particular countries or regions given the languages they speak and accent they have.
   • Determining that an object that is falling can be modelled with calculus.

   Cognition depends on training, the majority of people can learn to work through a problem in a particular way.

4.2.3 Question 2

library(tidyverse)
x <- 1:20
y <- x*3+2 + rnorm(length(x), 0, 15)
data <- data.frame(x,y)


ggplot(data, aes(x = x, y = y, col = y)) +
             geom_point() +
             guides(col = FALSE) +
             theme_classic() +
             stat_smooth(method = "lm", lty = 2, col = "purple") +
             labs(title = "Arbitrary time Series", x = "Time", y = "Response")

1. Human Visual System and Automated Analysis
   

   The pros and cons of Human Perception and Automated Analysis are best highlighted by considering the difficulties of unsupervised learning in the realm of Machine Learning.

   Unsupervised Machine Learning algorithms such as clustering and PCA (in 2 and 3 dimensions obviously) are very easily performed by humans, but implementing such algorithms automatically and analytically can be quite complex and in some cases resource intensive.

   This is very similar to the proximity Gestalt law discussed in question 8 below at 4.2.9.

   An advantage to the Human visual system is that the analysis can be done rapidly and requires no prior programming and maybe to a degree a human can be influenced quite rapidly in how the data is analysed or perceived whereas programming a machine learning algorithm or even a deductive algorithm can be complex and time consuming.

   The advantage to using Automated analysis and algorithms is that an algorithm:

   • is unbiased to the origins or nature of the data
   • can be extended to disjoint, large data that might have patterns outside what humans would be expected to recognise
   • can be extended to multiple dimensions
     • For example using KNN analysis might involve computing distances in higher dimensions
     • Or interpreting data with many variables, PCA can `flatten` out variables that don't significantly contribute to the variance of the response.
   1. Summary
      
      • Humans can make rapid and accurate relative judgements but these judements are

      susceptible to bias and influence.

      • Machines can make unbiased consistent judgements based on an algorithm (or

      atleast a machine learning algorithm) but cannot out of the box make a relative value judgement without human guidance.

4.2.4 Question 3

4.2.5 Question 4

1. (1) Number of Colours
   

   Humans can only identify about 8 colours on a plot, so try to use \(7\pm2\) and less than 10.

2. (2) Data Type
   

   Data may be ordered and may be continuous or discrete:

   	Ordered	Unordered
   Continuous	Colours should use a smooth divergent/sequential pallet	NA
   Discrete	Colours should use a small number of sequential colours	Colour

   pallet should be a small number of distinct colours |

   If data is continuous (and / or ordinal) then colours that follow a pattern should be used in order to illustrate that pattern, for example:

   library(RColorBrewer)
   library(tidyverse)
   # colorRampPalette(brewer.pal(9, "Blues"))(100) %>% plot()
   # my_cols <- brewer.pal(7,"Greens")
   # par(pty = "s", mai = c(0.1, 0.1, 0.4, 0.1))
   # display.brewer.pal(3,"Accent")
    display.brewer.pal(9,"Blues")

   

   If data is discrete then discrete and distinct colours should be used, if that data is also non-ordinal then those colours should avoid having a pattern, for example:

   library(RColorBrewer)
   library(tidyverse)
   # colorRampPalette(brewer.pal(9, "Blues"))(100) %>% plot()
   # my_cols <- brewer.pal(7,"Greens")
   # par(pty = "s", mai = c(0.1, 0.1, 0.4, 0.1))
   display.brewer.pal(3,"Accent")
   # display.brewer.pal(9,"Blues")

   

3. (3) Viewing Medium
   

   A visualisation may have different levels of perceived brightness depending on how it's viewed, for example a visualisation made on a pc with a black background (i.e. dark mode) may lead to a plot looking overly saturated, if that plot was printed the distinction between various factors may not be clear in graph.

4. (4) Contrast
   

   If a plot is converted to greyscale in order to print, there may be insufficient contrast to perceive differences, this can be avoided if attention is paid to ensure good 'seperation' between elements regardless of contrast.

5. (5) edge enhancement
   

   using colours to highlight edges may make a plot harder or easier to read, for example consider the following histograms:

   library(tidyverse)
   library(gridExtra)
   a <- rnorm(1000, 3, 300)
   b <- rbinom(length(a), size = 1, prob = 0.3)
   birthwt <- data.frame("A" = a, "B" = b )
   birthwt_pretty <- birthwt
   birthwt_pretty$B <- ifelse(birthwt$B, "A", "B")
   
   hist <- ggplot(birthwt_pretty, aes(x = A, fill = B,  y = ..density..)) +
            theme_classic() +
            labs(x = "Birth Weight", y = "Density") +
            guides(fill = guide_legend("Factor"))
   
   plots <- list()
   
   plots[[1]] <- hist + geom_histogram(position = "dodge2", col = "blue", binwidth = 300, lwd = 1)
   plots[[2]] <- hist + geom_histogram(position = "dodge2", col = "red", binwidth = 300, lwd = 1)
   plots[[3]] <- hist + geom_histogram(position = "dodge2", col = "black", binwidth = 300, lwd = 1)
   plots[[4]] <- hist + geom_histogram(position = "dodge2", binwidth = 300, lwd = 1)
   
   
   layout <- matrix(c(1:4), byrow = TRUE, nrow = 2)
   grid.arrange(grobs = plots, layout_matrix = layout)

   

   This effect can be made more pronounced when the plots have different scales an are arranged relative to each other:

   a <- rnorm(1000, 3, 300)
   b <- rbinom(length(a), size = 1, prob = 0.3)
   birthwt <- data.frame("A" = a, "B" = b )
   birthwt_pretty <- birthwt
   birthwt_pretty$B <- ifelse(birthwt$B, "A", "B")
   
   hist <- ggplot(birthwt_pretty, aes(x = A, fill = B,  y = ..density..)) +
            theme_classic() +
            labs(x = "Birth Weight", y = "Density") +
            guides(fill = guide_legend("Factor"))
   
   # hist + geom_histogram(position = "dodge2", col = "blue", binwidth = 300)
   
   plots <- list()
   
   # Dodge
   plots[[1]] <- hist + geom_histogram(position = "dodge2", col = "blue", binwidth = 300, lwd = 2)
   
   # Overlay
   plots[[2]] <-  hist + geom_histogram(binwidth = 300, col = "black", lwd = 2)
   
   # Single Histogram
   plots[[3]] <- hist + geom_histogram(binwidth = 300, col = "green", aes(group = 1), fill = "lightblue", lwd = 2)
   
   # Facet Grid
    plots[[4]] <- hist + geom_histogram(binwidth = 300, col = "purple", lwd = 2) +
            facet_grid(. ~ B) +
            guides(fill = FALSE)
   
   layout <- matrix(c(1, 1, 2, 3, 4, 4), byrow = TRUE, nrow = 3)
   # arrangeGrob(grobs = plots, layout_matrix = layout)
   grid.arrange(grobs = plots, layout_matrix = layout)

   

4.2.6 Question 5

1. Pokemon dashboard
   

   The Pokemon type dashboard uses colours effectively because it uses discrete and distinct colours to represent discrete variables.

   The colour choices are an intelligent choice of sufficiently distinct colours that are also differently related to the information to make the plot easy to interpret.

   The colours choses are also saturated enough to allow transperancy to be mapped to frequency and for the treemap to be sufficiently distinct.

   

2. Nuclear Waste Sites
   

   The Nuclear Waste Dashboard uses colour effectively because it uses a different colour pallette for the two types of plots which represent the number of sites in each category and the geographical location of any such site in that category.

   Tying a colour to a variable is important to visualise data without confusing the intended audience.

   

3. Neighbourhood Diversity
   

   The Neighbourhood Diversity visualisation uses colour effectively because the colour choses is a continous sequential pallete increasing in saturation, this makes interpreting continuous data easier because the pallete has a one-to-one correspondence with the density of the observation.

   

4. Using Colours
   

   The following graphic is an exemplar graphic from DataQuest but it shows how it is often desirable to restructure a visualisation based on the data as opposed to overusing colours which can be difficult to interpret in high amounts.

   

4.2.7 Question 6

4.2.8 Question 7

4.2.9 Question 8

1. Grouping
   
   • Objects close together will be perceived as groups
     • This is an example of humans performing something simple that is quite complex for machines.
2. Like Elements
   
   • If elements are similar in features such as shape, colour or size they will people are more ready to consider the data as grouped together
     • This means it can be important to use techniques such as blurring and desaturating visualisations in order to remove or enhance such distinctions as appropriate.
3. Connected Elements
   
   • Data That is connected in some way (by a line, curve or border) will give the effect of creating clusters or groups in data

4.2.10 Question 9

1. Quantitative Value
   

   If data has a quantitative value, the attributes that should be considered are

   1. Position
   2. Length
   3. Angle
   4. Area
   5. Volume
2. Nominal Value
   

   If data has a nominal value, the five most important attributes to consider are:

   1. Position of the Data
   2. Hue
   3. Texture
   4. Connection
   5. Containment

4.2.11 Question 10

5.4.1 Trees in Org Mode

1. TODO Make Headline Tree
   

   https://en.wikipedia.org/wiki/DOT_(graph_description_language)

   https://en.wikipedia.org/wiki/Graphviz

   https://orgmode.org/worg/org-tutorials/org-dot-diagrams.html

2. Mermaid
   

   1. Install mermaid.cli , refer to the GitHub:

      npm install -g mermaid.cli

   2. Install the Spacemacs layer by cloning this repo

      git clone https://github.com/arnm/mermaid-layer ~/.emacs.d/private/mermaid

      • If not using Spacemacs it may be necessary to install ob-mermaid and

      mermaid-mode from MELPA using M-x package-install. as described here.

   3. Add ob-mermaid to the org list here:

      (org-babel-do-load-languages
        'org-babel-load-languages
        '((R          . t)
          (latex       . t)
          (python      . t)
          (dot         . t)
          (mermaid     . t)
          (gnuplot     . t)
          (java        . t)
          (sed         . t)
          (shell       . t)
          (mathematica . t)
          (emacs-lisp  . t)))

   4. Add the mermaid layer to spacemacs

   5. Find out where the mermaid command line tool (i.e. mmdc) is installed:

      which mmdc
      sudo ln (which mmdc) /bin/mmdc

   6. Tell org-mode where mmdc is installed:

      (setq ob-mermaid-cli-path "/bin/mmdc")

   7. Test Mermaid using the following syntax

   1. Or another example
   1. Supported properties
      

      file - Output file. It should be either svg, png or pdf.

      width - Width of the page. Optional.

      height - Height of the page. Optional.

      theme - Theme of the chart, could be default, forest, dark or neutral. Optional.

      background-color - Background color. Example: transparent, red, '#F0F0F0'. Optional.

      mermaid-config-file - JSON configuration file for mermaid. Optional.

      css-file - CSS file for the page. Optional.

      pupeteer-config-file - JSON configuration file for puppeteer. Optional.

3. FlowchartJS
   
4. PlantUML
   

   I'm looking at this resource, there is also Official Documentation

   1. Add plantuml to the babel list

      1. annoyingly in vnote it is referred to as puml

      (org-babel-do-load-languages
       'org-babel-load-languages
       '((plantuml . t))) ; this line activates dot

   2. Download the latest plantuml.jar from here
      • Make sure to make it executable and to save it somewhere executable
   3. Install plantuml-mode mode from MELPA

   4. Add the following to your ~/.emacs.d/init.el

      (setq org-plantuml-jar-path (expand-file-name "/bin/plantuml.jar"))
      (add-to-list 'org-src-lang-modes '("plantuml" . plantuml))
      (org-babel-do-load-languages 'org-babel-load-languages '((plantuml . t)))

   5. Test the output